Providing information of tools by filtering image areas adjacent to or on displayed images of the tools

ABSTRACT

A system may comprise an image capture device to capture an image of a work site. The system may also comprise a processor to determine whether a tool disposed at the work site is energized and determine a first area of the captured image of the work site, including or adjacent to an image of a portion of the tool in the captured image. The processor may also determine a second area of the captured image including a remainder of the captured image that does not include the first area. Conditioned upon determining that the tool is energized, at least one of the first area and the second area of the captured image of the work site may be processed to indicate that the tool in the first area is being energized. The image of the work site with the first area and the second area may be displayed.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/768,062 (filed Feb. 15, 2013), which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention generally relates to robotic systems and inparticular, to a robotic system, and a method implemented therein, forproviding information of tools by filtering image areas adjacent to oron displayed images of the tools.

BACKGROUND OF THE INVENTION

In a robotic system, one or more tools may be telerobotically controlledby an operator to perform a procedure on an object at a work site. Acamera is provided at the work site to capture images of end effectorsof the tools as they interact with the object to perform the procedure,so that the operator may view their movements on a display whiletelerobotically controlling the tools using associated input devices.

During the procedure, it may be useful to provide the operator with toolinformation such as whether a tool is energized at the time or which ofa plurality of tools is energized at the time or which robotic arm isoperatively coupled to a specific tool at the time. The tool informationtypically may be provided as text or a graphic in an area on the displaythat is not within a current gaze area of the operator, such as in aboundary area circumscribing the display viewing area or off to one sideof the viewing area to avoid obscuring images of the end effectors andan object upon which the end effectors are performing a procedure at thetime.

When the tool information is provided outside the gaze area of theoperator, however, it may be distracting for the operator to visuallyfind and/or associate the provided tool information with itscorresponding tool because the operator's eyes must shift from the areain which the operator is currently gazing to another area on thedisplay. In the case of a stereo display, the situation becomes evenmore complicated, because the operator's eyes not only have to shiftvertically and horizontally around the display to find the toolinformation, they may also have to look for and focus on toolinformation at a different depth than the three-dimensional images ofthe object and tools that the operator is viewing at the time on astereo vision display.

OBJECTS AND BRIEF SUMMARY

Accordingly, one object of one or more aspects of the present inventionis a robotic system, and method implemented therein, that provides toolinformation within a gaze area of an operator as the operator is viewingan image of the tool on a display screen.

Another object of one or more aspects of the present invention is arobotic system, and method implemented therein, that provides toolinformation on a display that is easily associated to a tool beingviewed at the time by an operator.

These and additional objects are accomplished by the various aspects ofthe present invention, wherein the embodiments of the invention aresummarized by the claims that follow below.

Additional objects, features and advantages of the various aspects ofthe present invention will become apparent from the followingdescription of its preferred embodiments, which description should betaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing a roboticsystem utilizing aspects of the present invention.

FIG. 2 illustrates a front view of a movable cart usable in a roboticsystem utilizing aspects of the present invention.

FIG. 3 illustrates a perspective view of a tool usable in a roboticsystem utilizing aspects of the present invention,

FIG. 4 illustrates a front view of a console usable in a robotic systemutilizing aspects of the present invention.

FIG. 5 illustrates a flow diagram of a method for providing toolinformation on a display utilizing aspects of the present invention.

FIG. 6 illustrates a simplified view of a display in which an image ofan object is filtered according to the method of FIG. 5.

FIG. 7 illustrates a simplified view of a display in which an image ofan object is filtered according to the method of FIG. 5.

FIG, 8 illustrates a simplified view of a display in which an image ofan end effector of a tool is filtered according to the method of FIG. 5.

FIG. 9 illustrates a simplified view of a display in which an image ofan end effector of a tool is filtered according to the method of FIG. 5.

DETAILED DESCRIPTION

Although a medical robotic system is described herein, it is to beappreciated that the various aspects of the invention are not to belimited to medical robotic systems. They are applicable to roboticsystems in general.

FIG. 1 illustrates a top view of an operating room in which a medicalrobotic system 1000 is being employed by a Surgeon (“S”) to perform amedical procedure on a Patient (“P”). The medical robotic system in thiscase is a Minimally Invasive Robotic Surgical (MIRS) system including aConsole (“C”) utilized by the Surgeon while performing a minimallyinvasive diagnostic or surgical procedure on the Patient with assistancefrom one or more Assistants (“A”) while the Patient is on an Operatingtable (“O”).

The Console, as further described in reference to FIG. 4, includes aprocessor 43 which communicates with a movable cart 150 over a bus 110.A plurality of robotic arms 34, 36, 38 are included on the cart 150. Atool 33 is held and manipulated by robotic arm 36, another tool 35 isheld and manipulated by robotic arm 34, and an endoscope 37 is held andmanipulated by robotic arm 38. In this example, each of the tools 33, 35and the endoscope 37 is introduced through its own entry aperture in thePatient. As an example, tool 33 is inserted into aperture 166 to enterthe Patient.

The Surgeon performs the medical procedure by manipulating the inputdevices 41, 42 so that the processor 43 causes their respectivelyassociated robotic arms 34, 36 to manipulate their respective removablycoupled tools 33, 35 accordingly while the Surgeon views real-timeimages of a work site in three-dimensions (“3D”) on a stereo visiondisplay 45 of the Console. A stereoscopic endoscope 37 (having left andright cameras for capturing left and right stereo views) captures stereoimages of the work site. The processor 43 processes the stereo images sothat they may be properly displayed on the stereo vision display 45.

Each of the robotic arms 34, 36, 38 is conventionally formed of links,such as link 162, which are coupled together and manipulated throughactuatable joints, such as joint 163. Each of the robotic arms includesa setup arm and a slave manipulator. The setup arm positions its heldtool so that a pivot point occurs at its entry aperture into thePatient. The slave manipulator may then manipulate its held tool orendoscope so that it may be pivoted about the pivot point, inserted intoand retracted out of the entry aperture, and rotated about its shaftaxis. The robotic arms 34, 36, 38 may be carted into the operating roomvia the cart 150 or alternatively, they may be attached to sliders on awall or ceiling of the operating room.

FIG. 2 illustrates a front view of the cart 150. In addition to therobotic arms 34, 36, 38, shown in FIG. 1, a fourth robotic arm 32 isshown in FIG. 2. The fourth robotic arm 32 is available so that anothertool 31 may be introduced at the work site along with the tools 33, 35and endoscope 37.

FIG. 3 illustrates an exemplary tool 100 that may be used for eithertool 33 or 35. The tool 100 comprises an interface housing 108, a shaft104, an end effector 102, and a wrist mechanism 106 which includes oneor more wrist joints. The interface housing 108 is removably attached toa robotic arm so as to be mechanically coupled to actuators (such asmotors) in the slave manipulator of the attached robotic arm. Cables orrods, that are coupled to the actuators of the slave manipulator andextend through the shaft 104 from the interface housing 108 to the oneor more wrist joints of the wrist mechanism 106 and to the jaws of thetool's end effector 102, actuate the wrist joints and jaws in aconventional manner. The slave manipulator may also manipulate the toolin pitch and yaw angular rotations about its pivot point at the entryaperture, manipulate the tool in a roll angular rotation about thetool's shaft axis, and insert and retract the tool along a rail on therobotic arm as commanded by the processor 43.

FIG. 4 illustrates, as an example, a front view of the Console usable inthe medical robotic system 1000. The Console has left and right inputdevices 41, 42 which the user may grasp respectively with his/her leftand right hands to manipulate associated. devices, such as the tools 33,35, in preferably six degrees-of-freedom (“DOF”). Foot pedals 44 withtoe and heel controls are provided on the Console so the user maycontrol movement and/or actuation of devices associated with the footpedals. A processor 43 is provided in the Console for control and otherpurposes. The stereo vision display 45 is provided so that the user mayview the work site in stereo vision from images captured by thestereoscopic camera of the endoscope 37. Left and right eyepieces, 46and 47, are provided in the stereo vision display 45 so that the usermay view left and right two-dimensional (“2D”) display screens insidethe display 45 respectively with the user's left and right eyes.

The processor 43 performs various functions in the medical roboticsystem. One important function that it performs is to translate andtransfer the mechanical motion of input devices 41, 42 through controlsignals over bus 110 to command actuators of their associated roboticarms to actuate their respective joints so that the Surgeon caneffectively manipulate devices, such as the tools 33, 35, and endoscope37. Another function is to perform various methods described herein.Although described as a processor, it is to be appreciated that theprocessor 43 may be implemented by any combination of hardware, softwareand firmware. Also, its functions as described herein may be performedby one unit or divided up among different components, each of which maybe implemented in turn by any combination of hardware, software andfirmware. Further, although being shown as part of or being physicallyadjacent to the Console, the processor 43 may also comprise a number ofsubunits distributed throughout the system.

U.S. Pat. No. 6,659,939 B2 entitled “Cooperative Minimally InvasiveTelesurgical System,” which is incorporated herein by reference,provides additional details on a medical robotic system such asdescribed herein.

FIG. 5 illustrates a flow diagram of a method implemented in theprocessor 43 of the robotic system 1000 for providing tool informationon the display 45. In block 5001, the method determines the informationthat is to be displayed for the tools 33, 35. As an example, the toolinformation may include information of which of the tools iselectrically active or energized at the time for cauterization or otherpurposes. In the present example, it is determined that tool 33 has beenenergized. The determination may be made in this case using informationinteractively provided using conventional means by the operator of thesystem and/or information pre-programmed into the system.

In block 5002, the method determines the current pose (i.e., positionand orientation) of the tool 33 in its tool reference frame. Each toolis operatively coupled to a robotic arm that manipulates the toolaccording to control commands generated by the processor 43 in responseto operator manipulation of its associated input device. Themanipulation of the tool is relative to a pivot point, which serves asorigin for the tool reference frame. Determination of the current posefor each tool may be performed by using kinematics of the robotic armand/or other well known techniques. Additional details may be found, forexample, in U.S. 2006/0258938 A1 entitled “Methods and System forPerforming 3-D Tool Tracking by Fusion of Sensor and/or Camera Derived.Data during Minimally Invasive Robotic Surgery,” which is incorporatedherein by reference.

In block 5003, the method translates the determined tool pose in thetool reference frame to a tool pose in an image reference frame which isfrom the perspective of the stereo camera of the endoscope 37. As anexample, the tool pose in the tool reference frame may first betranslated to a tool pose in a fixed reference frame using a previouslydetermined transform for the tool reference frame to the fixedreference. The tool pose in the fixed reference frame may then betranslated to a tool pose in a camera reference frame using a previouslydetermined transform from the fixed reference frame to the camerareference frame. Finally, the tool pose in the camera reference framemay be translated to a tool pose in the image reference frame usingpreviously determined information of the camera pose in the camerareference frame. Additional details for such translations and transformsmay be found, for example, in U.S. Pat. No. 6,424,885 entitled “CameraReferenced Control in a Minimally Invasive Surgical Apparatus”, which isincorporated herein by reference.

In block 5004, the method determines an area of the image of the worksite which is to be filtered so as to indicate the tool 33 has beenenergized. The determination of the area to be filtered depends on anumber of factors including the current pose of the tool 33 and the typeof filtering to be employed to indicate the tool 33 is being energized.The area to be filtered is part, but not all of the image of the worksite.

In block 5005, the method filters the determined area of the work siteand displays the filtered image along with unfiltered images of theremainder of the work site in the display 45.

FIGS. 6-9 illustrates, as examples, simplified stereo views of thestereo vision display 45 from the perspective of an operator of thesystem 1000 after employing the method of FIG. 5 to provide toolinformation on the display 45. A viewing area 602 displays images whichhave been processed to provide telepresence from images of the work sitecaptured by the stereo camera of the endoscope 37. An optional boundaryarea 601 circumscribes the viewing area 602. In prior systems, toolinformation may be provided in the boundary area 601. The boundary area601, however, is generally outside a gaze area of the operator since theoperator's eyes are focusing on images of end effectors 333, 353 of thetools 33, 35 on the display 45 as the end effectors 333, 353 interactwith the object 610 at the work site to perform a procedure on theobject 610.

As examples of blocks 5004 and 5005, an area of the image of the object610, such as area 611 of FIG. 6, may be selected for enhancement if thetype of filtering is to enhance the image of the object 610 near oradjacent the tool 33 which is being energized at the time. In this case,the image of the remainder of the object 610, such as area 612 of FIG.6, is unfiltered (i.e., normal) along with images of the remainder ofthe work site. Enhancement in this case may comprise improving thecontrast, clarity, and/or brightness in the area. Alternatively, an areaof the image of the object 610, such as area 613 of FIG, 7, may beselected for non-filtering if the type of filtering is to degrade theimage of the object 610 that is not adjacent to the tool 33 which isbeing energized at the time. In this second case, the image of theremainder of the object 610, such as area 614 of FIG. 7, that is notadjacent to the tool 33, which is being energized at the time, isdegraded along with images of the remainder of the work site.Degradation in this case may include desaturation to remove colorcontent of the areas. As another example, the image of the end effectorof the tool being energized at the time may be selected if the type offiltering is to make the end effector appear to be glowing, such as theend effector 333 of tool 33 shown in FIG. 8. The image of the endeffector of the tool being energized at the time may also be selected ifthe type of filtering is make the end effector appear to have an aura,such as the end effector 333 of tool 33 shown in FIG. 9, In this case,the glow and/or aura may be temporary so that it fades away after thetool is initially energized or it may continue as long as the tool isbeing energized. Alternatively, or in addition to, filtering the imagesof the end effectors 333, 353 as described above, images of the wrists332, 352 and/or images of the shafts 331, 351 of the tools 33, 35 may befiltered in the same manner.

Although the various aspects of the present invention have beendescribed with respect to a preferred embodiment, it will be understoodthat the invention is entitled. to full protection within the full scopeof the appended claims.

1-12. (canceled)
 13. A system comprising: an image capture devicedisposed to capture an image of a work site; a display; and a processorconfigured to: determine whether a tool disposed at the work site isenergized; determine a first area of the captured image of the worksite, including or adjacent to an image of a portion of the tool in thecaptured image; determine a second area of the captured image, thesecond area including a remainder of the captured image that does notinclude the first area; and conditioned upon determining that the toolis energized, process at least one of the first area and the second areaof the captured image of the work site to indicate that the tool in thefirst area is being energized; and cause the image of the work site withthe first area and the second area to be displayed on the display. 14.The system of claim 13, wherein processing the at least one of the firstarea and the second area includes filtering the second area of thecaptured image of the work site to generate a filtered second area witha degraded effect that indicates that the tool in the first area isbeing energized.
 15. The system of claim 14, wherein causing the imageof the work site with the second area to be displayed includes causingthe filtered second area to be displayed.
 16. The system of claim 14,wherein the degraded effect includes a desaturation of color in thefiltered second area.
 17. The system of claim 13, wherein the portion ofthe tool comprises a working end of the tool.
 18. The system of claim13, wherein the processor is further configured to: determine whetherthe tool is energized by determining whether a working end of the toolis energized for performing electrocautery.
 19. The system of claim 13,wherein the image capture device comprises a stereo camera, wherein thedisplay is a stereo viewer, and wherein the image of the work site is astereo image of the work site.
 20. A method for providing toolinformation on a display, the method comprising: an image capture devicecapturing an image of a work site; a processor determining whether atool disposed at the work site is energized; the processor determining afirst area of the captured image of the work site, including or adjacentto an image of a portion of the tool in the captured image; theprocessor determining a second area of the captured image, the secondarea including a remainder of the captured image that does not includethe first area; conditioned upon determining that the tool is energized,the processor processing at least one of the first area and the secondarea of the captured image of the work site to indicate that the tool inthe first area is being energized; and the processor causing the imageof the work site with the first area and the second area to be displayedon the display.
 21. The method of claim 20, wherein processing the atleast one of the first area and the second area includes filtering thesecond area of the captured image of the work site to generate afiltered second area with a degraded effect that indicates that the toolin the first area is being energized.
 22. The method of claim 21,wherein causing the image of the work site with the second area to bedisplayed includes causing the filtered second area to be displayed. 23.The method of claim 21, wherein the degraded effect includes adesaturation of color in the filtered second area.
 24. The method ofclaim 20, wherein the portion of the tool comprises a working end of thetool.
 25. The method of claim 20, further comprising: the processordetermining whether the tool is energized by determining whether aworking end of the tool is energized for performing electrocautery. 26.The method of claim 20, wherein the image capture device comprises astereo camera, wherein the display is a stereo viewer, and wherein theimage of the work site is a stereo image of the work site.